SPEECH TRANSLATION

Neural Speech Translation using Lattice Transformations and Graph Networks (D19-53)

Challenge:	Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation.
Approach:	This tutorial introduces the techniques used in cutting-edge research on speech translation.
Outcome:	The proposed models achieve state-of-the-art performance with end-to-end speech translation for both high- and low-resource languages.

Copied to clipboard

Daniel Beck, Trevor Cohn, Gholamreza Haffari

Cross-Lingual Transfer Learning for Speech Translation (2025.naacl-short)

Challenge:	Existing work on end-to-end systems bypass the need for intermediate representations, but this approach is limited in practical applications.
Approach:	They propose a lattice-tosequence model which uses lattics as encoders and graph networks to address two problems by applying latticae transformations and a neural model.
Outcome:	The proposed model beats pipeline approaches while being orders of magnitude faster than previous work.

Copied to clipboard

Rao Ma, Mengjie Qian, Yassir Fathullah, Siyuan Tang, Mark Gales, Kate Knill

Towards Speech to Speech Machine Translation focusing on Indian Languages (2023.eacl-demo)

Challenge:	Increasing interest in building multilingual foundation models for NLP and speech research has led to limited data collection for training ST systems.
Approach:	They propose to use Whisper to explore the behavior of multilingual speech foundation models with restricted data.
Outcome:	The proposed model can translate to Chinese with a single language, and it can perform transcriptions in other languages.

Copied to clipboard

Vandan Mujadia, Dipti Sharma

Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023.findings-emnlp)

Challenge:	SSMT is a web application for translating videos from one language to another by cascading multiple language modules.
Approach:	They introduce an SSMT pipeline for translating videos from one language to another by cascading multiple language modules.
Outcome:	The proposed system can get 3.5+ MOS score for English to Hindi using human intervention.

Copied to clipboard

Arya McCarthy, Hao Zhang, Shankar Kumar, Felix Stahlberg, Ke Wu

SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020.emnlp-demos)

Challenge:	a challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations.
Approach:	They propose a large language model to split long ASR transcripts into segments that can be independently translated to maximize translation quality.
Outcome:	The proposed model improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 sets.

Copied to clipboard

Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, Juan Pino

Improve Speech Translation Through Text Rewrite (2025.coling-industry)

Challenge:	SimulEval is an evaluation toolkit for simultaneous text and speech translation.
Approach:	They propose a server-client scheme for simultaneous translation that uses server input and client policies to evaluate models.
Outcome:	The proposed evaluation toolkit is available for both text and speech translation.

Copied to clipboard

Jing Wu, Shushu Wang, Kai Fan, Wei Luo, Minpeng Liao, Zhongqiang Huang

ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)

Challenge:	Recent advances in speech translation (ST) research have focused on the unique characteristics of spontaneous speech, including accents and presentation quality.
Approach:	They propose to transform transcribed speech into a cleaner style more in line with the expectations of translation models built from written text.
Outcome:	Experiments on public and in-house translation models show that the proposed model can be effectively distilled into a standalone translation model.

Copied to clipboard

Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe

Consistent Transcription and Translation of Speech (2020.tacl-1)

Challenge:	ESPnet-ST is a new project for the quick development of speech-to-speech translation systems.
Approach:	They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines .
Outcome:	The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets.

Copied to clipboard

Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, Matthias Paulik

Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training (2024.findings-naacl)

Challenge:	Existing models that translate without transcribing focus on translation quality, while transcription receives less emphasis.
Approach:	They propose a method to evaluate consistency and compare different approaches . they propose 'coupled inference' models that feature a coupled inference procedure can achieve strong consistency.
Outcome:	The proposed model is poorly suited to the joint transcription/translation task, but is strong enough to train for consistency.

Copied to clipboard

Pavel Denisov, Thang Vu

CTC-based Compression for Direct Speech Translation (2021.eacl-main)

Challenge:	Recent advances in language modeling have led to the emergence of large language models capable ofvarious natural language processing tasks.
Approach:	They propose a multi-instructional training approach that integrates a large language model with a speech encoder to harness the capabilities of LLMs for speech recognition and beyond.
Outcome:	The proposed model can be trained and aligned with a multilingual LLM on 1900 hours of transcribed data from 139 languages.

Copied to clipboard

Marco Gaido, Mauro Cettolo, Matteo Negri, Marco Turchi

Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora (2022.aacl-short)

Challenge:	Existing studies have shown that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST).
Approach:	They propose a method which performs a phone-informed compression of the input audio in direct ST models by exploiting the Connectionist Temporal Classification (CTC) they demonstrate that their method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German)
Outcome:	The proposed method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) it reduces memory footprint by more than 10%, and is faster than previous approaches.

Copied to clipboard

Sara Papi, Alina Karakanta, Matteo Negri, Marco Turchi

Information-Transport-based Policy for Simultaneous Translation (2022.emnlp-main)

Challenge:	Existing models for subtitling require parallel data paired with audio inputs and textual translations.
Approach:	They propose to convert existing ST corpora into SubST resources without human intervention by exploiting audio and text in a multimodal fashion.
Outcome:	The proposed model achieves high segmentation quality in zero-shot conditions with manual and automatic segmentation.

Copied to clipboard

Shaolei Zhang, Yang Feng

Different Speech Translation Models Encode and Translate Speaker Gender Differently (2025.acl-short)

Challenge:	Simultaneous translation (ST) outputs translation while receiving source inputs . low latency restriction restricts ST to translating target tokens based on current received source tokens.
Approach:	They propose a system that outputs translation while receiving source inputs . it uses a read/write policy to decide whether to translate a target token or wait for the next source token .
Outcome:	The proposed model outperforms baselines and achieves state-of-the-art on text-to-text and speech-to text tasks.

Copied to clipboard

Dennis Fucci, Marco Gaido, Matteo Negri, Luisa Bentivogli, Andre Martins, Giuseppe Attanasio

MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder (2025.acl-industry)

Challenge:	Recent studies on interpreting the hidden states of speech models have shown their ability to capture speaker-specific features, including gender.
Approach:	They propose to use probing methods to assess gender encoding across ST models.
Outcome:	The proposed models capture speaker-specific features, including gender, while older models do not . low gender encoding capabilities result in systems’ tendency toward a masculine default, a translation bias that is more pronounced in newer architectures.

Copied to clipboard

Khai Le-Duc, Phuc Phan, Tan-Hanh Pham, Bach Phan Tat, Minh-Huong Ngo, Thanh Nguyen-Tang, Truong-Son Hy

Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)

Challenge:	Multilingual automatic speech recognition (ASR) in the medical domain is a critical foundational task, serving a wide range of downstream applications such as speech translation, spoken language understanding, and voice-activated assistants.
Approach:	They present the first multilingual medical ASR dataset and the first collection of small-to-large end-to end medical APR models spanning five languages: Vietnamese, English, German, French, and Mandarin Chinese.
Outcome:	The proposed model covers Vietnamese, English, German, French, and Mandarin Chinese, and is the first multilingual ASR dataset across five languages.

Copied to clipboard

Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino

SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022.emnlp-main)

Challenge:	Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations.
Approach:	They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition.
Outcome:	The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task.

Copied to clipboard

Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction (2025.emnlp-industry)

Challenge:	Existing models for pre-training text and speech are based on unlabeled audio data.
Approach:	They propose a unified-modal speech-unit-text pre-training model that connects speech encoders and text decoders with a shared unit encoder.
Outcome:	The proposed model improves on automatic speech recognition and speech translation tasks and achieves state-of-the-art performance on both the LibriSpeech ASR and MuST-C ST tasks.

Copied to clipboard

Victor Junqiu Wei, Weicheng Wang, Di Jiang, Yuanfeng Song, Lu Wang

End-to-End Speech Translation for Code Switched Speech (2022.findings-acl)

Challenge:	Automatic Speech Recognition (ASR) systems have a substantial number of erroneous recognition due to environmental noise, ambiguity, etc.
Approach:	They use a benchmark dataset to analyze ASR errors in the Chinese language . they then apply large language models to correct ASR error correction .
Outcome:	The proposed method is based on a dataset of ASR errors in the Chinese language . it shows prompting is not effective for ASR error correction .

Copied to clipboard

Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, Matthias Paulik

BIG-C: a Multimodal Multi-Purpose Dataset for Bemba (2023.acl-long)

Challenge:	Code switching (CS) is the phenomenon of interchangeably using words and phrases from different languages.
Approach:	They propose a new ST corpus that extends the joint transcription and translation setup.
Outcome:	The proposed model performs well even when no training data is used.

Copied to clipboard

Claytone Sikasote, Eunice Mukonde, Md Mahfuz Ibn Alam, Antonios Anastasopoulos

Speechformer: Reducing Information Loss in Direct Speech Translation (2021.emnlp-main)

Challenge:	Bemba is the most populous language of Zambia but lacks resources for research . despite its significance, Bemba remains under-resourced and lacking in high-quality data and resources for NLP experiments and language technologies.
Approach:	They propose a large multimodal dataset for Bemba that includes images, transcriptions and translations.
Outcome:	The proposed dataset is based on images, transcriptions and translations of Bemba speakers . it provides baselines on speech recognition, machine translation and speech translation tasks .

Copied to clipboard

Sara Papi, Marco Gaido, Matteo Negri, Marco Turchi

Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation (2022.acl-long)

Challenge:	Current approaches to speech-to-text translation (ST) use a pipeline of two sub-components - an automatic speech recognition (ASR) and a machine translation (MT) model.
Approach:	They propose an architecture that avoids initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria.
Outcome:	The proposed architecture achieves gains of up to 0.8 BLEU on the standard MuST-C corpus and up to 4.0 BLUE in a low resource scenario.

Copied to clipboard

Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, Marco Turchi

Is “moby dick” a Whale or a Bird? Named Entities and Terminology in Speech Translation (2021.emnlp-main)

Challenge:	grammatical gender languages are characterized by morphosyntactic chains of gender agreement marked on a variety of lexical items and parts-of-speech (POS).
Approach:	They propose to enrich the natural, gender-sensitive MuST-SHE corpus with two new linguistic annotation layers to explore gender bias.
Outcome:	The proposed models shed light on gender bias and its detection at several levels of granularity.

Copied to clipboard

Marco Gaido, Susana Rodríguez, Matteo Negri, Luisa Bentivogli, Marco Turchi

RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise (2022.findings-emnlp)

Challenge:	Among rare words, named entities and domain-specific terms are crucial . previous studies have neglected these important words due to limited options .
Approach:	They propose a benchmark to evaluate automatic translation systems for rare words . named entities and domain-specific terms are crucial for their translation .
Outcome:	The proposed benchmark is based on European Parliament speeches annotated with NEs and terminology.

Copied to clipboard

Jinming Zhao, Hao Yang, Gholamreza Haffari, Ehsan Shareghi

Whisper-UT: A Unified Translation Framework for Speech and Text (2025.emnlp-main)

Challenge:	Pre-trained speech Transformers in speech translation systems have facilitated state-of-the-art (SotA) results, but their computational cost is high.
Approach:	They propose a Reducer Adaptor block that could be seamlessly integrated within any Transformer-based speech encoding architecture.
Outcome:	The proposed Reducer Adaptor block outperforms the existing SotA architecture by an average of 0.68 BLEU score on 8 language pairs from Must-C.

Copied to clipboard

Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur

End-to-end ASR to jointly predict transcriptions and linguistic annotations (2021.naacl-main)

Challenge:	Encoder-decoder models have achieved remarkable success in speech and text tasks, but efficiently adapting them to diverse uni/multimodal scenarios remains a challenge.
Approach:	They propose a framework that leverages lightweight adapters to enable seamless adaptation across tasks.
Outcome:	The proposed framework improves speech translation performance through a 2-stage decoding strategy without requiring 3-way parallel data.

Copied to clipboard

Motoi Omachi, Yuya Fujita, Shinji Watanabe, Matthew Wiesner

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021.naacl-main)

Challenge:	Existing models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered.
Approach:	They propose a Transformer-based sequence-to-sequence model for automatic speech recognition that can produce high-quality transcriptions and linguistic annotations.
Outcome:	The proposed model can produce high-quality transcriptions and linguistic annotations on Japanese and English audio datasets.

Copied to clipboard

Siddharth Dalmia, Brian Yan, Vikas Raunak, Florian Metze, Shinji Watanabe

A Simple Concatenation can Effectively Improve Speech Translation (2023.acl-short)

Challenge:	ESPnet framework exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
Approach:	They propose a framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
Outcome:	The proposed framework outperforms the state-of-the-art on speech translation tasks by +6 and +3 BLEU on the two test sets of Fisher-CallHome and +4 BLUE on the English-German and English-French test sets.

Copied to clipboard

Linlin Zhang, Kai Fan, Boxing Chen, Luo Si

CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation (2023.findings-acl)

Challenge:	Experimental results show that in our unified cross-modal ST model, models can effectively utilize the auxiliary information from speech and text.
Approach:	They propose a unified cross-modal ST method which concatenates speech and text as the input and builds a teacher that can utilize both cross-modities simultaneously.
Outcome:	The proposed method can effectively utilize the auxiliary information from speech and text, and achieve compelling results on MuST-C datasets.

Copied to clipboard

Yikun Lei, Zhengshan Xue, Xiaohu Zhao, Haoran Sun, Shaolin Zhu, Xiaodong Lin, Deyi Xiong

Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021.acl-long)

Challenge:	End-to-end speech-totext translation (ST) data are limited due to the limited resources.
Approach:	They propose a knowledge distillation framework for speech translation that integrates knowledge from machine translation and decouples knowledge from non-target class knowledge.
Outcome:	The proposed framework outperforms state-of-the-art models on a benchmark dataset.

Copied to clipboard

Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, Jingbo Zhu

Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020.acl-main)

Challenge:	End-to-end Speech Translation (E2E ST) encoders lack global context representation, whereas MT encoder lacks it.
Approach:	They propose a Stacked Acoustic-and-Textual Encoding method for speech translation . they propose an adaptor module to alleviate representation inconsistency .
Outcome:	The proposed method achieves state-of-the-art BLEU scores of 18.3 and 25.2 on two ST tasks.

Copied to clipboard

Yu-An Chung, James Glass

Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)

Challenge:	Autoregressive coding targets are used to learn meaningful representations from unlabeled speech.
Approach:	They propose a method that trains an autoregressive RNN to generate an unseen future frame given a context such as recent past frames.
Outcome:	The proposed method can learn representations from unlabeled speech.

Copied to clipboard

Orion Weller, Matthias Sperber, Christian Gollan, Joris Kluivers

Phone Features Improve Speech Translation (2020.acl-main)

Challenge:	Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output .
Approach:	They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models.
Outcome:	The proposed model is statistically similar to cascading models, but has half the number of parameters.

Copied to clipboard

Elizabeth Salesky, Alan W Black

Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? (2021.acl-long)

Challenge:	End-to-end models for speech translation more tightly couple speech recognition (ASR) and machine translation (MT) compared to cascades, but performance gap remains in low-resource conditions .
Approach:	They propose two methods to incorporate phone features into current neural speech translation models.
Outcome:	The proposed models outperform existing models and cascades by up to 9 BLEU on low-resource conditions.

Copied to clipboard

Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, Marco Turchi

AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation (2021.findings-acl)

Challenge:	a gap between direct approaches to speech translation (ST) and traditional cascade solutions has gradually decreased . a recent study found that the subtle differences observed in their behavior are not sufficient for humans neither to distinguish them nor to prefer one over the other.
Approach:	They compare state-of-the-art systems representative of the two paradigms . they find subtle differences observed in their behavior are not sufficient .
Outcome:	The proposed system is compared with state-of-the-art systems representative of the two paradigms.

Copied to clipboard

Wuwei Huang, Dexin Wang, Deyi Xiong

Evaluating Gender Bias in Speech Translation (2022.lrec-1)

Challenge:	End-to-end speech translation models learn acoustic representations from the encoder, which is not desirable for cross-modal and cross-lingual translation.
Approach:	They propose an adaptive speech-to-text translation model that dynamically adapts acoustic states in the decoder.
Outcome:	The proposed model outperforms state-of-the-art speech translation models on two widely-used datasets.

Copied to clipboard

Marta R. Costa-jussà, Christine Basta, Gerard I. Gállego

The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023.findings-emnlp)

Challenge:	Existing evaluation techniques for gender biases are lacking in the field of machine translation.
Approach:	They propose to use a free evaluation set to evaluate gender bias in speech translation.
Outcome:	The proposed set is the speech version of WinoMT, an MT challenge set.

Copied to clipboard

Mutian He, Philip Garner

Does Joint Training Really Help Cascaded Speech Translation? (2022.emnlp-main)

Challenge:	Modern artificial intelligence is characterized by large pretrained language models with strong language capabilities to be adapted to various downstream tasks.
Approach:	They propose to use the task of speech translation (ST) to pretrain speech models for end-to-end SLU on intra- and cross-lingual scenarios.
Outcome:	The proposed model achieves higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks.

Copied to clipboard

Viet Anh Khoa Tran, David Thulke, Yingbo Gao, Christian Herold, Hermann Ney

AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs (2024.naacl-long)

Challenge:	Currently, in speech translation, the straightforward approach delivers state-of-the-art results, but fundamental challenges such as error propagation remain.
Approach:	They propose to combine a cascaded recognition system with a machine translation system to improve cascade speech translation.
Outcome:	The proposed methods can improve cascaded speech translation and suggest alternative training methods.

Copied to clipboard

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer

Mutual-Learning Improves End-to-End Speech Translation (2021.emnlp-main)

Challenge:	a new model for speech processing and reasoning uses curated data instead of text.
Approach:	They extend the instruction-tuned Llama-2 model with end-to-end speech processing and reasoning abilities without using any carefully curated paired data.
Outcome:	The proposed model outperforms or outperfects existing models on synthesized and recorded speech QA tests.

Copied to clipboard

Jiawei Zhao, Wei Luo, Boxing Chen, Andrew Gilman

Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021.acl-long)

Challenge:	Existing approaches to end-to-end speech translation (E2E) models only allow one way knowledge transfer, which is limited by the performance of the teacher model.
Approach:	They propose a one-way knowledge transfer paradigm where the MT and ST models are collaboratively trained and considered as peers rather than teacher/student.
Outcome:	The proposed model improves the performance of end-to-end speech translation (ST) task by combining knowledge from two models with peer models.

Copied to clipboard

Yun Tang, Juan Pino, Xian Li, Changhan Wang, Dmitriy Genzel

SDS-200: A Swiss German Speech to Standard German Text Corpus (2022.lrec-1)

Challenge:	Pretraining and multitask learning are widely used to improve the speech translation performance.
Approach:	They propose to train a speech translation model along with an auxiliary text translation task.
Outcome:	The proposed method improves translation quality by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs.

Copied to clipboard

Michel Plüss, Manuela Hürlimann, Marc Cuny, Alla Stöckli, Nikolaos Kapotis, Julia Hartmann, Malgorzata Anna Ulasik, Christian Scheller, Yanick Schraner, Amit Jain, Jan Deriu, Mark Cieliebak, Manfred Vogel

Wav2Prompt: End-to-End Speech Prompt Learning and Task-based Fine-tuning for Text-based LLMs (2025.naacl-long)

Challenge:	Using a web recording tool, participants were asked to translate their Swiss German text to their own dialect before recording it.
Approach:	They present a corpus of Swiss German dialectal speech with Standard German text translations . the dataset allows for training speech translation, dialect recognition, and speech synthesis systems .
Outcome:	The dataset allows for training speech translation, dialect recognition, and speech synthesis systems.

Copied to clipboard

Keqi Deng, Guangzhi Sun, Phil Woodland

Cross-modal Contrastive Learning for Speech Translation (2022.naacl-main)

Challenge:	Text-based large language models (LLMs) can be applied to a wide range of tasks without being explicitly trained.
Approach:	They propose a method which integrates spoken input with a text-based large language model (LLM) it takes LLM token embeddings as training targets and utilises a continuous integrate-and-fire mechanism for explicit speech-text alignment.
Outcome:	The proposed model can be applied to speech translation, speech understanding and spoken-query-based question answering tasks.

Copied to clipboard

Rong Ye, Mingxuan Wang, Lei Li

Semantic Text Processing

Challenge:	Existing approaches for speech translation focus on using additional data from MT and automatic speech recognition (ASR).
Approach:	They propose a cross-modal contrastive learning method for end-to-end speech-totext translation.
Outcome:	The proposed method outperforms existing methods on a popular benchmark MuST-C.

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)

Copied to clipboard

Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei

CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation (2023.findings-emnlp)

Challenge:	Existing work shows that pre-trained models can improve in various natural language processing tasks.
Approach:	They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data.
Outcome:	The proposed framework is superior to existing models on speech-to-text processing tasks.

Copied to clipboard

Xiaohu Zhao, Haoran Sun, Yikun Lei, Shaolin Zhu, Deyi Xiong

Semantic Text Processing

Challenge:	Existing speech-to-text translation models can extract features from speech inputs, but they may include non-linguistic speech factors such as pitch, timbre and speaker identity.
Approach:	They propose a content-centric speech representation disentanglement learning framework for speech translation that decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning.
Outcome:	The proposed framework outperforms state-of-the-art speech translation models and cascaded models on five translation directions.

Improving Speech Translation by Fusing Speech and Text (2023.findings-emnlp)

Copied to clipboard

Wenbiao Yin, Zhicheng Liu, Chengqi Zhao, Tao Wang, Jian Tong, Rong Ye

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)

Challenge:	In speech translation, multimodal data to address limitations of individual modalities has shown significant effectiveness.
Approach:	They propose a cross-modal model which supports three input modalities for speech, text and fused speech-text.
Outcome:	The proposed model achieves an average of 34.0 BLEU on MuST-C, GigaST and newstest benchmark.

Copied to clipboard

Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura

Representation Purification for End-to-End Speech Translation (2025.coling-main)

Challenge:	*LLaST* is a framework for building high-performance Large Language model based Speech-to-text Translation systems.
Approach:	They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems.
Outcome:	The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs.

Copied to clipboard

Chengwei Zhang, Yue Zhou, Rui Zhao, Yidong Chen, Xiaodong Shi

LibriVoxDeEn: A Corpus for German-to-English Speech Translation and German Speech Recognition (2020.lrec-1)

Challenge:	Existing approaches to enhance speech translation focus on enhancing knowledge transfer . factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer.
Approach:	They propose a framework that excludes content-agnostic perturbations from speech representations to mitigate their negative impact on ST.
Outcome:	The proposed framework significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a transcript-free setting.

Copied to clipboard

Benjamin Beilharz, Xin Sun, Sariya Karimova, Stefan Riezler

Contrastive Learning for Task-Independent SpeechLLM-Pretraining (2025.findings-acl)

Challenge:	a corpus of sentence-aligned triples of German audio, German text, and English translation is available for speech recognition . a large corpus is available to date for end-to-end speech translation based on parallel data .
Approach:	They present a corpus of sentence-aligned triples of German audio, German text, and English translation based on German audio books.
Outcome:	The proposed corpus is the largest resource for German speech recognition and for end-to-end German-to English speech translation.

Copied to clipboard

Maike Züfle, Jan Niehues

Semantic Text Processing

Challenge:	Large language models excel in speech processing tasks but their reliance on written text limits their application in real-world scenarios.
Approach:	They propose a task-independent speech pretraining stage and task-specific fine-tuning stage to adapt LLMs to speech processing tasks.
Outcome:	The proposed model outperforms models specialized on speech translation and question answering while being trained on 10% of the task-specific data.

End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation (2023.emnlp-main)

Copied to clipboard

Juan Pablo Zuluaga-Gomez, Zhaocheng Huang, Xing Niu, Rohit Paturi, Sundararajan Srinivasan, Prashant Mathur, Brian Thompson, Marcello Federico

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)

Challenge:	Conventional speech-to-text translation systems are trained on single-speaker utterances, but they may not be applicable to real-life scenarios where the audio contains conversations by multiple speakers.
Approach:	They propose a speaker-turn-aware conversational speech translation model that integrates automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format.
Outcome:	The proposed model outperforms the reference systems on the multi-speaker condition while attaining comparable performance on the single-speakspeaker conditions.

Copied to clipboard

Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, Yang Feng

End-to-End Simultaneous Speech Translation with Differentiable Segmentation (2023.findings-acl)

Challenge:	Existing simultaneous translation methods focus on text-to-text and speech-totext translation.
Approach:	They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning.
Outcome:	The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model.

Copied to clipboard

Shaolei Zhang, Yang Feng

Multilingual Parallel Corpus for Global Communication Plan (L18-1)

Challenge:	Existing methods to perform simultaneous speech translation always separate segmentation from the underlying model.
Approach:	They propose to use Differentiable Segmentation (DiSeg) to learn segmentation from the translation model.
Outcome:	Experimental results show that the proposed model can learn segmentation from the translation model.

Copied to clipboard

Kenji Imamura, Eiichiro Sumita

OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)

Challenge:	In this paper, we introduce the Global Communication Plan (GCP) Corpus . the corpus is sentence-aligned and covers ten languages, including many Asian languages .
Approach:	They introduce the Global Communication Plan (GCP) Corpus, a multilingual parallel corpus . it is sentence-aligned and covers ten languages, including many Asian languages .
Outcome:	The proposed corpus is sentence-aligned and covers ten languages, including many Asian languages.

Copied to clipboard

Yifan Peng, Yui Sudo, Muhammad Shakeel, Shinji Watanabe

CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation (2023.emnlp-main)

Challenge:	Autoregressive models can be slower during inference and have potential risks of hallucination.
Approach:	They propose an encoder-only speech foundation model based on Connectionist Temporal Classification.
Outcome:	The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification.

Copied to clipboard

Sathish Indurthi, Shamil Chollampatt, Ravi Agrawal, Marco Turchi

Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)

Challenge:	Cascaded approach is the most popular choice for speech translation, but lacks robustness when dealing with noisy inputs.
Approach:	They propose a cascaded approach that uses an automatic speech recognition model and a machine translation model to translate speech in one language to text in another language.
Outcome:	The proposed approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text.

Copied to clipboard

Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi

Leveraging Synthetic Targets for Machine Translation (2023.findings-acl)

Challenge:	a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics .
Approach:	They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context.
Outcome:	The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements.

Copied to clipboard

Sarthak Mittal, Oleksii Hrinchuk, Oleksii Kuchaiev

MELD-ST: An Emotion-aware Speech Translation Dataset (2024.findings-acl)

Challenge:	Using synthetic target data, training models on synthetic targets outperforms training on actual ground-truth data.
Approach:	They propose a recipe for training machine translation models on synthetic target data by leveraging a large pre-trained model.
Outcome:	The proposed model outperforms training on real-world translation datasets.

Copied to clipboard

Sirou Chen, Sakiko Yahata, Shuichiro Shimizu, Zhengdong Yang, Yihang Li, Chenhui Chu, Sadao Kurohashi

Simple and Effective Unsupervised Speech Translation (2023.acl-long)

Challenge:	Emotion plays a crucial role in human conversation.
Approach:	They present a MELD-ST dataset for the emotion-aware speech translation task . they show that fine-tuning with emotion labels can enhance translation performance .
Outcome:	The proposed dataset shows that fine tuning with emotion labels can improve translation performance in some settings.

Copied to clipboard

Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino

Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus (2020.acl-main)

Challenge:	Existing methods to train speech models without labeled data are limited for most languages.
Approach:	They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis.
Outcome:	The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions.

Copied to clipboard

Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, Marco Turchi

Effectively pretraining a speech translation decoder with Machine Translation data (2020.emnlp-main)

Challenge:	a growing number of studies have examined the issue of gender bias in speech translation . a gender bias is a systemic problem that reproduces gender stereotypes discriminating women.
Approach:	They present the first thorough investigation of gender bias in speech translation . they compare audio technologies for English-Italian/French translations .
Outcome:	The proposed method compares different technologies on two languages, English and French.

Copied to clipboard

Ashkan Alinejad, Anoop Sarkar

Lattice Transformer for Speech Translation (P19-1)

Challenge:	Existing approaches to improve the performance of AST systems are based on pretraining the encoder parameters using an ASR model, but using a pretrained MT decoder is not beneficial or improves the results.
Approach:	They propose to use an adversarial regularizer to bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities.
Outcome:	The proposed model can be pre-trained using the Automatic Speech Recognition (ASR) task even in different languages and improves in low resource settings.

Copied to clipboard

Pei Zhang, Niyu Ge, Boxing Chen, Kai Fan

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020.acl-main)

Challenge:	Recent advances in sequence modeling have highlighted the strengths of the transformer architecture.
Approach:	They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations.
Outcome:	The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task.

Copied to clipboard

Matthias Sperber, Matthias Paulik

Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection (2023.emnlp-main)

Challenge:	Until recently, the only feasible approach to translating acoustic speech signals into text was the cascaded approach.
Approach:	They propose a classification of the main challenges of traditional approaches to speech translation . they argue that end-to-end models fall short due to compromises made to address data scarcity .
Outcome:	This paper provides a brief survey of the main challenges of traditional approaches in speech translation . it reveals that many end-to-end models fail due to compromises made to address data scarcity.

Copied to clipboard

Dennis Fucci, Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa Bentivogli

DoCIA: An Online Document-Level Context Incorporation Agent for Speech Translation (2025.findings-acl)

Challenge:	Existing solutions to control speaker-related gender inflections in ST involve dedicated model retraining on gender-labeled data.
Approach:	They propose to use a gender-based inference-time solution to control speaker-related gender inflections in ST by replacing the implicitly learned internal language model with gender-specific external LMs.
Outcome:	The proposed approach outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms.

Copied to clipboard

Xinglin Lyu, Wei Tang, Yuang Li, Xiaofeng Zhao, Ming Zhu, Junhui Li, Yunfei Lu, Min Zhang, Daimeng Wei, Hao Yang, Min Zhang

Towards Zero-shot Learning for End-to-end Cross-modal Translation Models (2023.findings-emnlp)

Challenge:	Document-level context is crucial for speech translation due to noise from ASR . incorporating document-level contextual information into ST remains a challenge .
Approach:	They develop an online framework that integrates document-level context into machine translation . they use document-based modules to integrate document- level context into ST .
Outcome:	The proposed framework outperforms baselines in sentence and discourse metrics . it can correct ASR transcription errors and improve translation performance .

Copied to clipboard

Jichen Yang, Kai Fan, Minpeng Liao, Boxing Chen, Zhongqiang Huang

Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation (2023.findings-emnlp)

Challenge:	End-to-end zero-shot speech translation model is based on a zero-shot approach, but it is less competitive because of the limited amount of data available for multiple modalities.
Approach:	They propose an end-to-end zero-shot speech translation model that connects two pre-trained uni-modality modules via word rotator’s distance.
Outcome:	The proposed model performs better than or as well as those of the CTC-based models and can be trained in an end-to-end style to avoid error propagation.

Copied to clipboard

Hojin Lee, Changmin Lee, Seung-won Hwang

Soundwave: Less is More for Speech-Text Alignment in LLMs (2025.acl-long)

Challenge:	End-to-end approaches to speech translation suffer from data scarcity compared to machine translation (MT).
Approach:	They propose a method which combines knowledge distillation and consistency learning to break the dilemma of learning-forgetting.
Outcome:	The proposed method outperforms the previous methods on a MuST-C dataset even without additional data.

Copied to clipboard

Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, Haizhou Li

Improving Language and Modality Transfer in Translation by Character-level Modeling (2025.acl-long)

Challenge:	Existing end-to-end speech large language models rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth.
Approach:	They propose a training strategy and a novel architecture to address representation space gap and sequence length inconsistency in speech and text.
Outcome:	The proposed model outperforms other advanced speech LLMs in speech translation and AIR-Bench speech tasks with only a fraction of the training data.

Copied to clipboard

Ioannis Tsiamas, David Dale, Marta R. Costa-jussà

SpeechQE: Estimating the Quality of Direct Speech Translation (2024.emnlp-main)

Challenge:	Current translation systems cover only 5% of the world's languages . expanding to the long-tail of low-resource languages requires data-efficient methods that rely on cross-lingual and cross-modal knowledge transfer.
Approach:	They propose a character-based approach to improve adaptability to new languages and modalities by using a teacher-student approach and parallel translation data to obtain a SONAR character-level encoder.
Outcome:	The proposed model outperforms subword-based models in speech-to-text translation on the FLEURS benchmark on 33 languages and achieves state-of-the-art generalizability to unseen languages.

Copied to clipboard

HyoJung Han, Kevin Duh, Marine Carpuat

Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair (2024.emnlp-main)

Challenge:	Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored.
Approach:	They propose a new quality estimation system based on cascaded and end-to-end architectures.
Outcome:	The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation.

Copied to clipboard

Yusuke Sakai, Mana Makinae, Hidetaka Kamigaito, Taro Watanabe

What Has LeBenchmark Learnt about French Syntax? (2024.lrec-main)

Challenge:	Existing siMT corpora are limited due to high costs and limited annotator capabilities.
Approach:	They propose a method to convert ST corpora into interpretation-style corpors by fine-tuning models with Large Language Models.
Outcome:	The proposed method reduces latency while achieving better quality compared to other models.

Copied to clipboard

Zdravko Dugonjić, Adrien Pupier, Benjamin Lecouteux, Maximin Coavoux

LCMA-SRT: Language-Conditional Mixture-of-Experts Adapters for Joint Multilingual Speech Recognition and Translation (2026.acl-long)

Challenge:	Pretrained acoustic models are increasingly used for downstream speech tasks such as automatic speech recognition, speech translation, spoken language understanding or speech parsing.
Approach:	They propose to probing a pretrained acoustic model for French for syntactic information using the Orféo treebank.
Outcome:	The proposed model is trained on 7k hours of spoken French and obtained reasonable results on tasks that require higher level linguistic knowledge.

Copied to clipboard

Nanjie Li, Xiaoyong Guo, Hao Huang, Xu Haihua, Wei Shi